Make native parser the default by ilevkivskyi · Pull Request #21823 · python/mypy

ilevkivskyi · 2026-08-09T17:41:26Z

For now the PR only contains updates for different error messages in the new parser. As I started working on making native parser the default, I discovered there are still several small bugs/inconsistencies.

I will start fixing them one by one in separate PRs, but you are also welcome to fix some if you have time, just let me know before you start working on them.

There is one question to discuss for now: it looks like there is no support in new parser for type comments in for and with statements. I vaguely remember this is intentional. What is our stance on this? Do we declare type comments for for and with deprecated and thus only supported by the old parser?

cc @JukkaL @hauntsaninja

hauntsaninja

Do we declare type comments for for and with deprecated and thus only supported by the old parser?

This seems totally fine to me. It doesn't affect public interfaces so there isn't going to be much fallout from losing these types.

…rser

ilevkivskyi · 2026-08-19T14:08:44Z

@JukkaL I completely forgot that some tests were already skipped with _no_native_parse. So in fact we have a bit more things to fix. Fortunately, it looks like all of them are easy. But this still means a bit more work is needed.

ilevkivskyi · 2026-08-19T14:10:41Z

(I will start with all the fixes that require changes in ast_serialize, and will handle the mypy-only things later)

#78) This fixes panics found in python/mypy#21823

ilevkivskyi · 2026-08-24T17:23:48Z

@JukkaL I have been looking into test failures, and one test case looks weird. I think the current behavior in testOverloadIfSplitFunctionDef (with the old parser) is wrong, while the behavior with the new parser is actually correct (note there is # flags: --always-true True so that behavior for f1 and f3 should be identical). Just wanted to double-check I am not missing something.

This matches behavior of the new parser to the old one. Not adding tests, since some existing tests fail when switched to new parser, see #21823

Old parser has some special treatment for docstrings. Match that in the new parser. Not adding tests, since some existing tests fail when switched to new parser, see #21823

ilevkivskyi · 2026-08-25T10:28:20Z

@JukkaL Few more things for our discussion today:

Ruff parser has a problem with parsing string that contain unicode surrogates, they all get replaced with the same replacement/placeholder symbol, see Surrogate code points are internally represented as U+FFFD REPLACEMENT CHARACTER astral-sh/ruff#13666. This is not a problem for mypy, but it is a problem for mypyc, as it affects runtime behavior, thus testUnicodeSurrogate fails. For some reasons fixing this is non-trivial, so RustPython (Rust-based Python interpreter that also uses Ruff parser) has an ~1000 lines patch that uses WTF, see Parse surrogates in string literals properly RustPython/RustPython#5629
There are some cases where new parser errors are worse (either more verbose or more generic) than old parser errors. I am not going to spend much time on this, unless you think this is important.
New parser does not error on some invalid function/variable type comments, and simply ignores them. I am not sure what to do about that, but I guess it may be fine.
New parser ignores invalid # type: ignore comments instead of giving an error. I think this is an important thing to decide, potentially we may want to replicate old behavior exactly.
New parser forcefully adds escapes even when they are not needed, for example b"what's up" is parsed as b"what\'s up". TBH I don't really like this, what is the motivation for this?

…rser

ilevkivskyi · 2026-09-07T15:17:48Z

Oh, it looks like there are 4 more issues (previously unnoticed because they appear only in some CI jobs):

Version check for *-unpacking is completely broken, we allow foo[*x] on Python 3.10, but prohibit Callable[[*Ts], int] which is fine
There is something wrong with newlines in f-strings on Windows
There are couple more edge cases where new parser ignores type comments silently instead of giving an error
This one looks like a bug in Ruff parser: it allows un-parenthesized tuples as exception types (which is a SyntaxError in Python)

All except the Windows thing look like easy fixes, I will try to do this today and make another ast-serialize release.

danielhollas · 2026-09-07T21:59:47Z

This one looks like a bug in Ruff parser: it allows un-parenthesized tuples as exception types (which is a SyntaxError in Python)

This is allowed since python 3.14 per PEP-758

ilevkivskyi · 2026-09-07T22:03:16Z

@danielhollas Oh, thanks! I will make the check conditional on Python version.

This reverts commit fde2f8c.

This reverts commit 322bfe5.

…rser

github-actions · 2026-09-08T16:44:58Z

Diff from mypy_primer, showing the effect of this PR on open source code:

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/utilities/collections.py:504: error: "None" object is not iterable  [misc]
+ src/prefect/utilities/collections.py:505: error: "None" object is not iterable  [misc]
- src/prefect/_internal/concurrency/api.py:181: error: "Call" expects no type arguments, but 1 given  [type-arg]
- src/prefect/_internal/concurrency/api.py:181: error: Variable "prefect._internal.concurrency.api.T" is not valid as a type  [valid-type]
+ src/prefect/_internal/concurrency/api.py:182: error: Variable "prefect._internal.concurrency.api.T" is not valid as a type  [valid-type]
- src/prefect/_internal/concurrency/api.py:181: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ src/prefect/_internal/concurrency/api.py:182: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ src/prefect/_internal/concurrency/api.py:183: error: "Call" expects no type arguments, but 1 given  [type-arg]
+ src/prefect/_internal/concurrency/api.py:183: error: Variable "prefect._internal.concurrency.api.T" is not valid as a type  [valid-type]
+ src/prefect/_internal/concurrency/api.py:183: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases

colour (https://github.com/colour-science/colour)
- colour/characterisation/aces_it.py:975: error: Ellipses cannot accompany other parameter types in function type signature  [syntax]
+ colour/characterisation/aces_it.py:976: error: Unexpected "..."  [misc]
+ colour/characterisation/aces_it.py:977: error: Unexpected "..."  [misc]

antidote (https://github.com/Finistere/antidote)
- src/antidote/lib/interface_ext/__init__.py:1359: error: "staticmethod" expects 2 type arguments, but 1 given  [type-arg]
+ src/antidote/lib/interface_ext/__init__.py:1361: error: "staticmethod" expects 2 type arguments, but 1 given  [type-arg]
- src/antidote/lib/interface_ext/__init__.py:1359: error: Missing type arguments for generic type "staticmethod"  [type-arg]
+ src/antidote/lib/interface_ext/__init__.py:1361: error: Missing type arguments for generic type "staticmethod"  [type-arg]

Tanjun (https://github.com/FasterSpeeding/Tanjun)
- tanjun/commands/menu.py:420: error: Type argument "_UserCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:422: error: Type argument "_UserCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
- tanjun/commands/menu.py:420: error: Type argument "_UserCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:422: error: Type argument "_UserCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
- tanjun/commands/menu.py:420: error: Type argument "_MessageCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:424: error: Type argument "_MessageCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
- tanjun/commands/menu.py:420: error: Type argument "_MessageCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:424: error: Type argument "_MessageCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]

pandera (https://github.com/pandera-dev/pandera)
+ tests/pandas/test_model.py:170: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/pandas/test_model.py:172: error: Invalid type comment or annotation  [valid-type]
+ tests/pandas/test_model.py:172: note: Cannot use a function call in a type annotation
+ tests/pandas/test_model.py:282: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/pandas/test_model.py:283: error: Invalid type comment or annotation  [valid-type]
+ tests/pandas/test_model.py:283: note: Cannot use a function call in a type annotation
+ tests/pandas/test_model.py:285: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/pandas/test_model.py:286: error: Invalid type comment or annotation  [valid-type]
+ tests/pandas/test_model.py:286: note: Cannot use a function call in a type annotation
+ tests/pandas/test_model.py:290: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/pandas/test_model.py:291: error: Invalid type comment or annotation  [valid-type]
+ tests/pandas/test_model.py:291: note: Cannot use a function call in a type annotation
+ tests/pandas/test_model.py:293: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/pandas/test_model.py:294: error: Invalid type comment or annotation  [valid-type]
+ tests/pandas/test_model.py:294: note: Cannot use a function call in a type annotation
+ tests/pandas/test_model.py:296: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/pandas/test_model.py:297: error: Invalid type comment or annotation  [valid-type]
+ tests/pandas/test_model.py:297: note: Cannot use a function call in a type annotation

ibis (https://github.com/ibis-project/ibis)
- ibis/selectors.py:434: error: Variable "ibis.common.collections.frozendict" is not valid as a type  [valid-type]
+ ibis/selectors.py:437: error: Variable "ibis.common.collections.frozendict" is not valid as a type  [valid-type]
- ibis/selectors.py:434: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ ibis/selectors.py:437: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
- ibis/expr/types/generic.py:152: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
+ ibis/expr/types/generic.py:164: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
- ibis/expr/types/generic.py:152: note: Perhaps you need "Callable[...]" or a callback protocol?
+ ibis/expr/types/generic.py:164: note: Perhaps you need "Callable[...]" or a callback protocol?
- ibis/expr/types/generic.py:170: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
+ ibis/expr/types/generic.py:171: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
- ibis/expr/types/generic.py:170: note: Perhaps you need "Callable[...]" or a callback protocol?
+ ibis/expr/types/generic.py:171: note: Perhaps you need "Callable[...]" or a callback protocol?
- ibis/expr/types/temporal.py:428: error: "Value" expects no type arguments, but 1 given  [type-arg]
+ ibis/expr/types/temporal.py:431: error: "Value" expects no type arguments, but 1 given  [type-arg]
- ibis/expr/types/temporal.py:1161: error: "Value" expects no type arguments, but 1 given  [type-arg]
+ ibis/expr/types/temporal.py:1174: error: "Value" expects no type arguments, but 1 given  [type-arg]

static-frame (https://github.com/static-frame/static-frame)
+ static_frame/core/frame.py:7192: error: Unused "type: ignore" comment  [unused-ignore]
+ static_frame/core/frame.py:7193: error: Need type annotation for "name"  [var-annotated]
+ static_frame/core/frame.py:7435: error: Unused "type: ignore" comment  [unused-ignore]
+ static_frame/core/frame.py:7436: error: Need type annotation for "name"  [var-annotated]

apprise (https://github.com/caronc/apprise)
- apprise/plugins/fluxer.py:512: error: Unsupported operand types for + ("str" and "bool")  [operator]
+ apprise/plugins/fluxer.py:513: error: Unsupported operand types for + ("str" and "bool")  [operator]
- apprise/plugins/fluxer.py:512: note: Right operand is of type "bool | str"
+ apprise/plugins/fluxer.py:513: note: Right operand is of type "bool | str"
- apprise/plugins/fluxer.py:512: error: Incompatible types in assignment (expression has type "Any | str", target has type "bool")  [assignment]
+ apprise/plugins/fluxer.py:513: error: Incompatible types in assignment (expression has type "Any | str", target has type "bool")  [assignment]

ilevkivskyi · 2026-09-08T16:47:26Z

OK, there is just one issue left, the CRLF failure on Windows. The problem is that when reading a source file in Python, it has this concept of universal newlines, so that e.g. \r\n is automatically translated into \n, while if we do this in Rust using std::fs::read_to_string() we get the original newlines. In most cases this doesn't matter, except when we compile multi-line strings in mypyc. In this case Ruff parser preserves the newlines in the string literals as well, so we (incorrectly) get

repr("""foo
bar""") == "'foo\r\nbar'"

on Windows. It seems to me we cannot fix the newlines for string literals only, because we will need to preserve the escaped ones, handle f-strings and t-strings, and it all gets quite tricky. The only feasible solution I see is to replace all newlines in the source code before sending it to the Ruff parser. @JukkaL what do you think?

ilevkivskyi · 2026-09-08T19:00:42Z

@JukkaL Btw I have found a crate that does some (very) fast newline normalization https://crates.io/crates/newline_normalizer (it however has very few downloads).

[WIP] Make native parser the default

d81aad4

This comment has been minimized.

Sign in to view

Also update parse error test output

049c5fe

This comment has been minimized.

Sign in to view

hauntsaninja reviewed Aug 10, 2026

View reviewed changes

ilevkivskyi added 2 commits August 18, 2026 18:49

Merge remote-tracking branch 'upstream/master' into default-native-pa…

9511035

…rser

Re-enable and update some more tests

86a897a

This comment has been minimized.

Sign in to view

ilevkivskyi mentioned this pull request Aug 20, 2026

Fix panics while searching type comments in function with syntax error mypyc/ast_serialize#78

Merged

ilevkivskyi added a commit to mypyc/ast_serialize that referenced this pull request Aug 21, 2026

Fix panics while searching type comments in function with syntax error (

969b630

#78) This fixes panics found in python/mypy#21823

This was referenced Aug 25, 2026

Preserve docstrings in the new parser #21890

Merged

Match new parser behavior w.r.t. posonly args #21891

Merged

JukkaL pushed a commit that referenced this pull request Aug 25, 2026

Match new parser behavior w.r.t. posonly args (#21891)

cea9058

This matches behavior of the new parser to the old one. Not adding tests, since some existing tests fail when switched to new parser, see #21823

ilevkivskyi added 2 commits August 25, 2026 13:41

Merge remote-tracking branch 'upstream/master' into default-native-pa…

a74105a

…rser

Fix args parsing; update couple tests

6bebea2

This comment has been minimized.

Sign in to view

Fix/update some more tests

68f2051

This comment has been minimized.

Sign in to view

ilevkivskyi added 2 commits September 2, 2026 19:41

Merge remote-tracking branch 'upstream/master' into default-native-pa…

d7c282e

…rser

Update few more tests

1108dda

This comment has been minimized.

Sign in to view

ilevkivskyi mentioned this pull request Sep 3, 2026

2.4 Release Planning #21935

Open

ilevkivskyi added 2 commits September 6, 2026 15:24

Merge remote-tracking branch 'upstream/master' into default-native-pa…

199f613

…rser

Update some more tests

938a82a

This comment has been minimized.

Sign in to view

This was referenced Sep 6, 2026

Errors from new parser can be worse (compared to old parser) #21947

Open

New parser deviates from old parser in edge cases with function type comments #21948

Open

Reject invalid type ignores mypyc/ast_serialize#83

Merged

Update some parse error tests (run on 3.10 only)

1ec9337

This comment has been minimized.

Sign in to view

Merge remote-tracking branch 'upstream/master' into default-native-pa…

b81937c

…rser

ilevkivskyi changed the title ~~[WIP] Make native parser the default~~ Make native parser the default Sep 7, 2026

Update more tests (hopefully just one more left)

0cbdf19

This comment has been minimized.

Sign in to view

Fix star-unpack version check

d60dcdb

This comment has been minimized.

Sign in to view

Try running parse tests on current Python version

e445f9c

This comment has been minimized.

Sign in to view

Debug Windows newline issue (sorry for the noise)

322bfe5

This comment has been minimized.

Sign in to view

[WIP] Compare with interpreted mode

fde2f8c

This comment has been minimized.

Sign in to view

ilevkivskyi added 4 commits September 8, 2026 17:21

Revert "[WIP] Compare with interpreted mode"

41da5d8

This reverts commit fde2f8c.

Revert "Debug Windows newline issue (sorry for the noise)"

432e7cb

This reverts commit 322bfe5.

Merge remote-tracking branch 'upstream/master' into default-native-pa…

2d85a55

…rser

Skip couple more tests

443151a

ilevkivskyi mentioned this pull request Sep 8, 2026

Simple-minded universal newline normalization mypyc/ast_serialize#90

Open

Uh oh!

Uh oh!

Conversation

ilevkivskyi commented Aug 9, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

hauntsaninja left a comment

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi commented Aug 19, 2026

Uh oh!

ilevkivskyi commented Aug 19, 2026

Uh oh!

This comment has been minimized.

ilevkivskyi commented Aug 24, 2026

Uh oh!

ilevkivskyi commented Aug 25, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ilevkivskyi commented Sep 7, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

danielhollas commented Sep 7, 2026

Uh oh!

ilevkivskyi commented Sep 7, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

github-actions Bot commented Sep 8, 2026

Uh oh!

ilevkivskyi commented Sep 8, 2026

Uh oh!

ilevkivskyi commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants